Computerized devices and computer programs control almost every aspect of our life, from writing documents to controlling traffic lights through commerce, communication, industry and almost any other area. An important area of computer programming relates to large-scale computationally heavy programs used for scientific processing and research, which process a lot of input data, perform numerous calculations and output results. Such programs may be related, for example, to various types of number crunching, climate changes analysis and weather forecasting, genomic research, machine learning, and other areas.
Designing and programming such programs present developers with many challenges, some of which are related to the large volumes of processed data and the processing complexity. Such programs usually require the collaborative work of multiple programmers or programmer groups, over long periods of time.
In some environments, such programs are developed in parts, wherein each programmer or group is responsible for a part of the processing, sometime referred to as a block. The blocks then have to be combined to provide solutions to the required tasks.
Current technologies do not provide a satisfactory solution to combining such blocks. The combination requires feasible transference of work products from one block to the other, so that a data-producing block generates the data in the same format as required by a block which receives the information. In addition, one or more blocks may require to process their input again, but may not be able to store all that input, or such storing may be too expensive in performance or resources terms. Current technologies do not provide adequate solution to such limitation as well.
Additional requirements beyond designing the basic correspondence between consecutive block, relate to rate balancing between the blocks. Such requirements may relate, for example, to eliminating overflow between blocks when an information-producing block generates information in a rate higher than the information-consuming block can process, balancing the accumulation of different inputs required for each block, or the like.